Right, I have ordered some new temperature and humidity sensors. That mean I have to write some new code, but that is not the question.
I want to log (and possibly do some control) temperature, RH and solar data. I will probably have a minimum of 4 sensors for each, so 12 in all (may stick some more temp ones on as temperature is so useful).
The question is, should I write just one bit of Python Script that runs from start to finish i.e. reads the temp sensors, then the RH ones, then the solar ones, then the RTC and then write that to a file, or write 3 separate scripts for each and then use a 4th one to pull all the data together and write to a log file.
The first one is my preferred choice as I can probably do that, but do not know if it is the best way to do it.
The same applies to any outputs, should I incorporate them in a script and act on them as I go along or wait till all the data is collected and then flick the switches as needed?

Thanks Ed. As I will probably be ditching the 1-Wire stuff for this project, it is probably easiest to keep it in one file. Shall see when the DHT11s get here how easy they are to read.

There are a couple of bits of code on the internet about making them work on the RPi, one one is Python and the other is C.
I don't really want to try and understand C (tried that in 1986) but may have to.

What is a kernal-space driver. I think I came across something about a driver for these sensors.

From the little reading I have done about it, existing code seems to keep reading until the flag and/or the CRC give the correct result and then returns that line as correct. Shall have a better idea when I get them. I may just use the existing C stuff and then use the Python to log the data from it.

Pete
That would be useful, thank you.
Not got mine yet, they said they would take between 7 and 21 days.

Thanks Pete
I am still waiting for them to be delivered, so shall report back as soon as I have them wired up.
One thing that I may need to change is the GPIO pin as I have a real time clock fitted and can't remember which pins it uses.

This is more about Python than the RPi.
I have a large CSV file and I want to get useful data out of it. Excel is a bit slow, so how easy is it to write a bit of script to open the file and do some conditional 'sums' on it.
Basically it starts with a date, then the data in a few columns. I want to get the usual statistical stuff out, mean, median, mode, standard deviation, count, skew, kurtosis etc.
I need it to be pretty easy to understand.
Thanks

Thanks Ed, I think I follow most of that, shall see if I can get something out of my data later, looks like I am in for a wet weekend.

Bit more of a Python tutorial than is really appropriate to the forum but I think one or two others than just Steamy might have some interest:

Posted By: SteamyTeaGoing to see if I can understand this bit of code, I shall put my comments before the code:

#opens the csv file so that all lines can be read. if I put a number in the () after the readlines would that limit the lines it reads, either to or from?
for line in open('EGPC_HDD_12.0C.csv').readlines():

Not quite. Actually readlines reads all of the lines from the file and puts them in a list. If you had a *really* big file (bigger than the GB or so of memory you've probably got to hand) then a different approach would be needed. The for loop then loops over the whole of that list.

Because readlines returns a list you can index it as you please. For example, to read just the 11th, 12th and 13th lines use:

for line in open('EGPC_HDD_12.0C.csv').readlines()[10:12]: (oops, changed the example text but didn't change the code)
for line in open('EGPC_HDD_12.0C.csv').readlines()[10:13]:

bearing in mind that the first line of the file is line zero and that array slices go from the first mentioned index up to but not including the second.

To read the last 10 lines of the file:

for line in open('EGPC_HDD_12.0C.csv').readlines()[-10:]:

The alternative approach if you want to deal with very large files would be to leave out the readlines() bit:

for line in open('EGPC_HDD_12.0C.csv'):

Then you get an iterator over the lines and only the one you're processing at the moment is in memory (plus a small bit of buffering hidden in the i/o system). You can't then do the indexing I mentioned above.

#creates a working file somewhere in memory that is used and manipulated, in this case stripping out unwanted stuff
line = line.strip()

Yes. Removes whitespace from the beginning and end of the line. In particular, removes the newline character which readlines leaves on the end of most lines.

#only works on lines that have more than 1 character in them, but don't understand what the (line[0] in '12') does, what does the [0] and '12' do?
if (len(line) >= 1) and (line[0] in '12'):

A string is an indexable list of characters. line[0] selects the first character in the line in the same way an index like this would select the first element in a list or tuple. The "in '12'" part tests to see if the first character in the line is in the string '12', ie, if it's a '1' or a '2' so a line from the second or third millenniums AD.

We need to check for at least one character in the line first otherwise the line[0] bit will blow up on empty lines.

#creates columns that were separated by a comma and gives them names
date, hdd, estimate = line.split(',')

Yes. Also checks there are exactly three columns - if there were more or fewer it would blow up.

#makes the data in the hdd column have a floating decimal point
hdd = float(hdd)

Not quite. The date, hdd and estimate column values split out of the line are strings of characters. Using the float type as a function like this causes it to convert that string to a number.

#all about the date format, I assume that map is some clever thing that puts parts of the date that are separated by a - into a form that does not have the -, but a space instead and only prints the integer. So if my date and time info is in the form 27/07/2014 01:10 then I will have to split it into two, the date and the time, then recombine the date part and the time part.
y, m, d = map(int, date.split('-'))

date.split('-') returns a list of strings: the three separate fields which make up the date. Map is a function which takes a function and a list (or other iterable) and applies the function to each element of the list making up a new list of the results. As with float above, int called as a function takes a string and returns an integer value parsed from it.

(Map can also take multiple iterables but let's not go there.)

#prints out the result with a blank space between each element.
print(y, m, d, hdd)

Yep.

Luckily this is Python, not mathematics. There's no line 11.999 (or platform 9¾). Having [m:n] mean from m up to but not including n is convenient as you know you'll get (n-m) elements back (if m and n are positive and the addressed elements are present).

It also allows special cases like extracting zero elements at the start of a list with [0:0] whereas if the end was included that would extract one element and to get zero you'd have to write [0:-1] which wouldn't work because negative indexes count back from the end of the list so that'd actually give you all but the last element (which is what the index -1 references):

>>> ('a', 'b', 'c', 'd')[0]
'a'
>>> ('a', 'b', 'c', 'd')[-1]
'd'
>>> ('a', 'b', 'c', 'd')[0:-1]
('a', 'b', 'c')
>>> start = 2
>>> length = 2
>>> ('a', 'b', 'c', 'd')[start:start+length]
('c', 'd')

Why would you want to extract zero elements? Maybe it's just a special case. Say you're splitting a CSV line and the first field is empty then the line would start with a comma. Rather than having to do special code for this case you can just remember that the field length is zero and extract away:

Also, the poor old parentheses are already horribly overworked in Python being used for the normal mathematical grouping in expressions, for parameter lists and for tuples. Overloading them for indexing as well would probably result in a nasty accident.

Ed Davies asserted: "Luckily this is Python, not mathematics."

Well, yes, and I regard my primary knowledge as computing rather than mathematics. But I still think that Python convention is very counter-intuitive and flies in the face of convention:

http://en.wikipedia.org/wiki/Comparison_of_programming_languages_%28array%29#Slicing

I confess I'm prejudiced. I learnt to hate languages using tab at the start of lines with make and I'm probably not going to overcome that hatred enough to ever love Python.

OK, thank you for that summary. I have saved it and will work thro' step by painful step starting with the link to the quick start guide. It will take me weeks and I may (!) need some prompts along the way. Perhaps some of the content of this thread will clarify as I proceed.

Posted By: SteamyTeaLinux file system is just horrible

Oh no, can't let that pass. Unix file systems are a thing of beauty. Microsoft/DOS files systems are just horrible

So after 30 years of finding an .exe file to start a program, generally in the 'program files' directory, I have still not worked out how to do that in Linux.

That's because you're expecting things to be as difficult as they are on Windows Steamy! Generally on Linux you don't need to know where the individual files are. A competent OS should know this kind of thing so you don't have to! You either click an icon or launch it from a command line.

For example, the command to launch Firefox from the command line is:

firefox

No need to know where the executable is at all, just type that from any location.

If you're just running scripts on an RPi you might need to click on or run individual executables from their install location, but that works exactly the same way as Windows. Just go to wherever you dumped the application (/home? /opt?) to and run it. On the command line you do:

./script_name