Creating a csv file with measurement data and meta data in the first line

Question

I want to save measurement data I receive via the CAN bus in a csv file. Apart from the measurement data I want to also store some meta data I need to convert the measurement data I receive via CAN.

The measurement data is for three axis and has 1024 values each. Due to timing issues I store the data in a list while I receive it and store it after the transmission of measurement data I finished. For the reception of the can messages I use the python-can library. The meta data I receive in the first message with measurement data.

I want to store everything in one file with a format like this

x data	y data	z data	meta data 1	meta data 2
x(0)	y(0)	z(0)	2000	3
x(1)	y(1)	x(1)
...	...	...
x(1022)	y(1022)	z(1022)
x(1023)	y(1023)	z(1023)

I am struggling with writing the meta data and first bin of the measurement data in the first line. After writing the first bin and meta in the first line the cells for the meta data should be empty.

There is probably an easy solution to that but I am not searching with the right keywords.

Currently I am just writing the complete list into the csv with the write row function

with open(path, 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(["x_data", "y_data", "z_data"])
    for msgData in msgBuffer:
        writer.writerow([((msgData[1] <<8) | msgData[2]), ((msgData[3] <<8) | msgData[4]) , ((msgData[5] <<8) | msgData[6])])

Here I am struggling with just writing the first line of measurement data and the meta data and then proceed with just writing the measurement data.

@MichaelM. I am getting the meta with a message indicating the start of a measurement data transmission. I am sending multiple messages. 1. message indicating start of measurement data transmission 2. 1024 messages with measurement data 3. end of transmission message To save the data I use the code in my post. While the transmission is happening I just append all messages to a list. msgBuffer.append(msg.data) — melfie, Apr 01 '23 at 17:53
@MichaelM. At the moment I am storing the measurement and meta data in seperate files in the same folder. I can extract the meta data and it is saved to a variable. The content of that variable I want to write to the first line of the csv. — melfie, Apr 01 '23 at 17:59

score 0 · Accepted Answer · answered Apr 01 '23 at 18:12

I'm not entirely sure how you're getting your metadata, or how to access it, but you've said that it's saved in a variable. For purposes of this answer, I'll assume there are all the metadata is in a list, metadata (for example: [2000, 3]).

Right now, you're only writing a header file with the rows for your data, but you'll also want to include the headers for the metadata. To future-proof this, you can create a generator expression that creates just enough columns for however many elements are in metadata. Then, you can change your for-loop to use enumerate() to check if this is the first iteration or not. If it is not, you can expand metadata into the line, otherwise you can fill it with nothing:

import csv

msgBuffer = [[1, 2, 3, 4, 5, 6, 7], [7, 8, 9, 10, 11, 12, 13]]  # just example data
metadata = [2000, 3]  # just example data

with open('test.csv', 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(["x_data", "y_data", "z_data", *(f'meta data {i+1}' for i in range(len(metadata)))])
    for i, msgData in enumerate(msgBuffer):
        data = metadata if i == 0 else ['' for i in range(len(metadata))]
        writer.writerow([((msgData[1] <<8) | msgData[2]), ((msgData[3] <<8) | msgData[4]) , ((msgData[5] <<8) | msgData[6]), *data])

This will create a CSV file that looks like this:

x_data,y_data,z_data,meta data 1,meta data 2
515,1029,1543,2000,3
2057,2571,3085,,

Creating a csv file with measurement data and meta data in the first line

1 Answers1