Saving/storing data into CSV file(s)

t is quite handy to store (some) data on your IoT device. Simplest way to do this is to write data into standardised CSV (comma separated values) format. There are slight variations of this format, but the main features are that data is separated into rows by adding a ‘\n’ (newline), and separation between columns is done by ‘,’ (comma) or ‘;’ (semicolon).

A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. The use of the comma as a field separator is the source of the name for this file format. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields. The CSV file format is not fully standardized. The basic idea of separating fields with a comma is clear, but that idea gets complicated when the field data may also contain commas or even embedded line breaks. CSV implementations may not handle such field data, or they may use quotation marks to surround the field. Quotation does not solve everything: some fields may need embedded quotation marks, so a CSV implementation may include escape characters or escape sequences.

Therefore you have the structure of the file like:

id;value1;value2;value3
1;12;"abc";21
14;88;"md ";142

Writing data to (csv) file

There are multiple libraries for handling .csv files in python, but the simplest way for writing data into csv file is to use standard Python file handling.

We should do this in 3 steps:

Open file in a+ mode (for appending) csv_file = open('some_filename.csv', 'a+')
Write data to file (don’t forget to write the '\n' newline/EndOfLine character) csv_file.write('Some data separated by ; (semicolons)' + '\n')
Close file (only once you close data that you wanted to write gets written to the file!) csv_file.close()

Example: Writing sensor data to csv

Here we will modify example of reading data from Rotary Angle sensor on A0 by incorporating the data writing to csv

import grovepi
import time

### (Optional) CSV header writing part BEGINNING ###

# Define filename of the csv file that you want to use
output_filename = 'angle_sensor_data.csv' #in case the name is fixed, every time when you run the script, you will be using the same file

# Create header line (like a header of the table)
header = ("Timestamp","Angle","Sensor_value") # define array of header values (names of columns)
strHeader = ';'.join(header) # Join all data into one string separated by ; (as required for CSV)

# Write header line to the file 
csv_file = open(output_filename, 'a+') # Open in 'a+' mode, new data is appended to the end of the file
csv_file.write(strHeader + '\n') # Finish the line using '\n' character
csv_file.close() # Closes the file for further operations (and actually writes data to the file)

### CSV header writing part END ###

# Connect the Grove Rotary Angle Sensor to analog port A0
# SIG,NC,VCC,GND
potentiometer = 0

grovepi.pinMode(potentiometer,"INPUT")

# Allow 1 sec for board to initialise 
time.sleep(1)

# Reference voltage of ADC is 5v
adc_ref = 5

# Vcc of the grove interface is normally 5v
grove_vcc = 5

# Full value of the rotary angle is 300 degrees, as per it's specs (0 to 300)
full_angle = 300

while True:
    try:
        # Read sensor value from potentiometer
        sensor_value = grovepi.analogRead(potentiometer)

        # Calculate voltage
        voltage = round((float)(sensor_value) * adc_ref / 1023, 2)

        # Calculate rotation in degrees (0 to 300)
        degrees = round((voltage * full_angle) / grove_vcc, 2)

        print("sensor_value = %d voltage = %.2f degrees = %.1f" %(sensor_value, voltage, degrees))

### CSV writing part BEGINNING ###
        # Format angle and sensor_value as a csv file line
        data = [str(time.time()), str(degrees), str(sensor_value)] # Define array/list of string values
        strData = ';'.join(data) # Join all data into one string separated by ; (as required for CSV)

        # Write sensor values to the csv file
        csv_file = open(output_filename, 'a+') # Again open (the same) file in 'a+' mode
        csv_file.write(strData + '\n') # Write string of joined (; separated) data & finish the line using '\n' 
        csv_file.close() # Close the file again

### CSV writing part END ###

    except Exception as e:
        
        # Output error in case exception arises 
        print ("Error: " + str(e))

    # Wait 1 second before reading next measurement 
    time.sleep(1)

And after running this code, you will get a file called angle_sensor_data.csv (in the same folder as your python script).

You can completely remove the header writing part, as it is not necessary, but it makes a difference once you import file to Excel for example and convert it to table.