Adding data to INF

TAPs für die SRP- und CRP-Serie
DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

Adding data to INF

#1

Beitrag von DeltaMikeCharlie » Mo 13. Mai 2013, 06:59

Greetings from Australia, apologies for posting in English.

I have written a TAP that saves the EPG data when recordings are made. Normally, I save this data to a separate file, but I want to add a feature so that the data can be archived (to external media) with the MPG(REC)/INF/NAV without having to worry about archiving my other file.

Adding the extra data to the INF would seem like a logical choice because then any TAP or the standard firmware could copy the files without knowing about my extra file.

For the purposes of my TAP, I could just add my data and be finished. But I thought it might be nice to cooperate with other TAPs that may need to store information in the INF file as well, so I would like to ask for opinions on the following proposal:

First, some background: Research on my Australian TMS PVR suggests that there are 3 types of INF file.

0 snapshots = 3,320 bytes (3k)
1 snapshot = 132,636 bytes (132k)
4 snapshots = 499,572 (499k)

I have found that if I write extra data after 499k, the firmware ignores this extra data, the PVR does not change or destroy the extra data.

In addition to this, I have found that if I have a short (3k or 132k) INF, if I fill the space up to 499k with NULL characters, the PVR still works fine and will even add 4 snapshots without damaging the extra data beyond 499k.

Step 1: As a pre-requisite to writing extra data, the INF needs to be padded to 499k.

When it comes to writing the extra data, I would like to do it in a way that other TAPs could also use if they had data to store.

Step 2: I propose that each extra piece of data should be preceded by a fixed-length header. This header would have enough information for a TAP to either recognise the following data as its own or some other TAP's and provide the ability to skip over it.

The INF file would then contain: <INF DATA> + <HEADER><TAP 1 DATA> + <HEADER><TAP 2 DATA>, etc.

Actually, the <DATA> is optional if there is a small amount of data and it can be saved in the header fields.

Here is the struct that I am considering for the header.

Code: Alles auswählen

typedef struct TYPE_EXTINF {
  char   magic[4];        //TFr+
  dword  payloadSize;     //Size of the data payload to follow.
  dword  tapID;           //ID of the TAP that created the header, and following data.
  dword  type;            //Flag for the TAP to note the record type.
  dword  tag;             //Option for the TAP to save an additional tag for the data.
  dword  timeStamp;       //Time stamp of when the record was written.
  byte   secondsStamp;    //Seconds from the previous time stamp.
  word   version;         //Version number of the data block.
  byte   deleted;         //Flags if this record is logically deleted and should be ignored.
  char   lang[3];         //Language of the extensible data.
  char   description[32]; //Human-readable description of the data type.
} TYPE_EXTINF;
Using a fixed-length header and the payloadSize field, it should be easy for a TAP to skip data that is not relevant to it.

The magic field is very important for detecting corruption. If an INF is larger than 499k, but the first 4 extra bytes after 499k are not magic, the INF is corrupted in some way. This repeats for every header, if a header is read and the magic field is wrong, then there is a corruption.

Extending an INF becomes simple.

1) If < 499k, pad to 499k with NULLs, if already >= 499k then seek EOF.
2) Write a header.
3) Write some data.
4) Close the file.

Reading is simple too:

1) seek 499k+1.
2) Read header.
3) Check magic.
4) Check TAP ID.
5) Read data or seek next header with payloadSize.

Feedback would be appreciated, especially if anyone has written a TAP that could benefit from adding information to the INF.

Best Regards,
DeltaMikeCharlie.

Benutzeravatar
Twilight
Zauberküchencheflehrling mit extra Butter
Zauberküchencheflehrling mit extra Butter
Beiträge: 64936
Registriert: Fr 9. Dez 2005, 09:17
Receivertyp: 1 x SRP 2100(TMS) TFIR und .1 x SRP 2410 M
Wohnort: Wien Umgebung

AW: Adding data to INF

#2

Beitrag von Twilight » Mo 13. Mai 2013, 07:09

we also discussed to add some more extra data in the inf files, so we have to think about that we don't destroy the data from each other and can use both :thinker:

twilight

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#3

Beitrag von DeltaMikeCharlie » Mo 13. Mai 2013, 08:55

I agree, it is good that we discuss this so that we don't break each other's TAPs and all can work together.

Have you already built some code to do this, or is it in the planning stages?

Do you have any research you can share? Are European INF files the same size as Australian INF files, for example? Do you have native INF files any bigger than 499k?

Benutzeravatar
Twilight
Zauberküchencheflehrling mit extra Butter
Zauberküchencheflehrling mit extra Butter
Beiträge: 64936
Registriert: Fr 9. Dez 2005, 09:17
Receivertyp: 1 x SRP 2100(TMS) TFIR und .1 x SRP 2410 M
Wohnort: Wien Umgebung

AW: Adding data to INF

#4

Beitrag von Twilight » Mo 13. Mai 2013, 09:08

that is something we have planed. baybe it is a good idea to add this into the firebird lib.
i think firebird knows if there are differences between australian and european inf files :thinker:

twilight

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#5

Beitrag von DeltaMikeCharlie » Mo 13. Mai 2013, 09:44

Once again, I agree.

If this is useful and agreed by everyone, FireBird could choose to add it to the library so that we can all use it and be assured that we are all using the same compatible code.

First the developer community needs to agree that: 1) We need it; and 2) How we do it.

Benutzeravatar
FireBird
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Beiträge: 28969
Registriert: Fr 9. Dez 2005, 09:59
Receivertyp: SRP-2401CI+ TFIR
vu+ Duo 4k
Wohnort: Wien

AW: Adding data to INF

#6

Beitrag von FireBird » Mo 13. Mai 2013, 12:29

1) DMC has said that he needs it; and 2) Twilight has said how we do it. :D

I can offer to hack some functions which allow saving arbitrary data into the inf.

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#7

Beitrag von DeltaMikeCharlie » Mo 13. Mai 2013, 13:33

FireBird,

Have you already agreed on a method in Europe, or have you only discussed it in general terms?

If you have already agreed a method, I'm happy for you to write some code based on that method.

If you have only discussed it, I would like to try and write some code based on my proposal and see how it turns out. Assuming that you agree with the overall strategy that I have proposed.

Also, do you know if European PVRs create any INFs different to the 3 types that I have described above at the start of the thread? (3k, 132k & 499k)

Benutzeravatar
FireBird
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Beiträge: 28969
Registriert: Fr 9. Dez 2005, 09:59
Receivertyp: SRP-2401CI+ TFIR
vu+ Duo 4k
Wohnort: Wien

AW: Adding data to INF

#8

Beitrag von FireBird » Mo 13. Mai 2013, 14:28

My idea was similar but with a minimum header. I think that we should keep the size of the inf small and that it is a wasted space if a TAP doesn’t care about time stamps or version numbers. I think that we should leave that to the TAP and its payload.

Code: Alles auswählen

char   magic[4];              //TFr+
word   SigLen;
word   payloadSize;           //Size of the data payload to follow.
char   Sig[SigLen];           //A unique name to address this payload
byte   Payload[payloadSize];
I'm not aware of different inf sizes.

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#9

Beitrag von DeltaMikeCharlie » Mo 13. Mai 2013, 22:35

Let's go with your approach and expand it if we find we need to. I look forward to seeing your code.

I too thought about a minimal header approach at first, and if this was in RAM I would have stayed with that approach. Because we have a multi-gigabyte disk to play with, I decided to add some more details into the header.

I do think that you need to consider adding the TAP ID to the header so that you don't need to interpret a variable length Sig to discover if it is your data and decide if to skip or not. If Header.TAPID != My_TAPID then Skip SigLen+PayloadSize.

I also think that your code should not assume that the INF is under the /DataFiles directory structure. The function should accept a full path to the INF. Nor should you assume that the extension is ".inf".

For example, in my TAP (the one that started this discussion) I create a hard link to the INF in my /ProgramFiles/Settings directory, this is where I would like to extend my INF from so that I don’t need to try and discover the "proper" file name and location.

I can send you samples of the 3 Australian INF formats if you need them to test with.

Benutzeravatar
FireBird
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Beiträge: 28969
Registriert: Fr 9. Dez 2005, 09:59
Receivertyp: SRP-2401CI+ TFIR
vu+ Duo 4k
Wohnort: Wien

AW: Adding data to INF

#10

Beitrag von FireBird » Mo 13. Mai 2013, 22:50

DeltaMikeCharlie hat geschrieben:Because we have a multi-gigabyte disk to play with, I decided to add some more details into the header.

The problem is that some (firmware) code might assume that the inf is small and always load the whole file into the RAM.
I do think that you need to consider adding the TAP ID to the header so that you don't need to interpret a variable length Sig to discover if it is your data and decide if to skip or not. If Header.TAPID != My_TAPID then Skip SigLen+PayloadSize.
That would make it impossible to share data between different TAPs. Your INFplus is a good example of where I do not want to restrict the access to INFplus only. I don?t think that we will have a performance issue because these functions are not the ones which will be called a hundred times a second.
I also think that your code should not assume that the INF is under the /DataFiles directory structure.
My idea was to use the same approach as the TAP API: current directory.
The function should accept a full path to the INF. Nor should you assume that the extension is ".inf".

For example, in my TAP (the one that started this discussion) I create a hard link to the INF in my /ProgramFiles/Settings directory, this is where I would like to extend my INF from so that I don?t need to try and discover the "proper" file name and location.
I would have added a .inf if it wasn?t part of the file name but this is a good point so I?ll leave it away.

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#11

Beitrag von DeltaMikeCharlie » Mo 13. Mai 2013, 23:33

The problem is that some (firmware) code might assume that the inf is small and always load the whole file into the RAM.
This is true, but I think that this is more an issue for the payload than the header. In an extreme example, what happens if a TAP is created to add multi-megabyte screen snaphots for every 10 minutes of recording? The augmented INF could be huge.

Does the library impose a limit on the amount of data that can be added to an INF? If so, how do we determine that limit?
That would make it impossible to share data between different TAPs. Your INFplus is a good example of where I do not want to restrict the access to INFplus only. I don’t think that we will have a performance issue because these functions are not the ones which will be called a hundred times a second.
Sorry, my example was unclear. The library would not do the TAP ID comparison, the TAP would make the comparison. The library would return everything to the TAP and let the TAP decide.
My idea was to use the same approach as the TAP API: current directory.
Current directory is a good idea.
I would have added a .inf if it wasn’t part of the file name but this is a good point so I’ll leave it away.
Agreed.

deangelj
Erfahrener Benutzer
Erfahrener Benutzer
Beiträge: 156
Registriert: Mo 12. Mär 2007, 21:21
Receivertyp: TRF-2400
Wohnort: Sydney, Australia

AW: Adding data to INF

#12

Beitrag von deangelj » Di 14. Mai 2013, 00:14

[quote="DeltaMikeCharlie"]
I do think that you need to consider adding the TAP ID to the header so that you don't need to interpret a variable length Sig to discover if it is your data and decide if to skip or not. If Header.TAPID != My_TAPID then Skip SigLen+PayloadSize.
[/quote]


Hey DMC, I would like to add Series Recording Info to the .INF. Now this may be used by the writer TAP, as well as other TAPs that may want to look at that info. I do agree that you need a unique identifier so that you can find the relevant data. TAPID sounds good but it could be anything. The other issue is that TAPs don't require a TAPID but I guess we can make that assumption if they want to use this service.
Zuletzt geändert von deangelj am Di 14. Mai 2013, 00:19, insgesamt 1-mal geändert.

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#13

Beitrag von DeltaMikeCharlie » Di 14. Mai 2013, 00:38

JDA, I see your point.

It would still be nice to have some kind of unique identifier that would allow a reader TAP to quickly determine if the data is relevant without having to read the Sig or the Payload.

Pragmatically speaking, there are not a lot of TAP developers and we do tend to talk amongst ourselves in our little community anyway. What about a char[4] to store a tag and we just do our best to avoid duplicating? Setting up an official registry of augmented INF tags would be overkill.

FireBird, On the size of the INF issue: I plan to build a multi-gigabyte dummy augmented INF on my development PVR today and see what happens.

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#14

Beitrag von DeltaMikeCharlie » Di 14. Mai 2013, 01:24

I admit that this was only 1 test on 1 machine that may yet prove to be statistically insignificant.

I expanded a 499k INF to be 3,702,961,608 bytes (3.44Gb) and loaded it onto my development PVR.

With MemoryWatcher_TMS running I played the associated recoding and pressed "Info" and saw the correct info with no spike in memory use. I also pressed "Info" from the standard file list with the same result. I also tested the file manager that I use, TMSArchive, also with no ill effect.

Not a comprehensive test, but an encouraging result.

Benutzeravatar
FireBird
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Beiträge: 28969
Registriert: Fr 9. Dez 2005, 09:59
Receivertyp: SRP-2401CI+ TFIR
vu+ Duo 4k
Wohnort: Wien

AW: Adding data to INF

#15

Beitrag von FireBird » Di 14. Mai 2013, 08:43

Maybe the name “Sig” was misleading. It is a unique name which identifies a data block. The reader TAP just needs to know the name, it will never see the other data blocks. For example, INF+ wants to write two EPG events into the inf, then it will call

infData_Set(inffilename, "INF+_Event_1", data);
infData_Set(inffilename, "INF+_Event_2", data);

A reader TAP is now able to read all event data until the function returns false because the named block doesn’t exist:

infData_Get(inffilename, "INF+_Event_1", &data) = true
infData_Get(inffilename, "INF+_Event_2", &data) = true
infData_Get(inffilename, "INF+_Event_3", &data) = false

Therefore the writer needs to document the data structure and the structure names.
Zuletzt geändert von FireBird am Di 14. Mai 2013, 08:46, insgesamt 1-mal geändert.

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#16

Beitrag von DeltaMikeCharlie » Di 14. Mai 2013, 09:14

OK, so "Sig" is the tag! Much clearer now.

Benutzeravatar
FireBird
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Beiträge: 28969
Registriert: Fr 9. Dez 2005, 09:59
Receivertyp: SRP-2401CI+ TFIR
vu+ Duo 4k
Wohnort: Wien

AW: Adding data to INF

#17

Beitrag von FireBird » Di 14. Mai 2013, 19:20

The current FBLib now contains a new block of infData functions. Some quick tests looked good.



bool infData_isAvail(char *infFileName, char *NameTag, dword *PayloadSize);

Checks if a data block with the NameTag is available in the inf file.
  • infFileName (in): the name of the file, which needs to be in the current directory.
  • NameTag (in): the unique name of the data block.
  • PayloadSize (out, optional): receives the size of the payload or 0 if the data block doesn’t exist. NULL may be passed if the value is of no interest.
  • Return value: TRUE if the specified data block exists.

bool infData_Get(char *infFileName, char *NameTag, dword *PayloadSize, byte **Payload);

Retrieves the payload of a specific data block.
  • infFileName (in): the name of the file, which needs to be in the current directory.
  • NameTag (in): the unique name of the data block.
  • PayloadSize (out, optional): receives the size of the payload or 0 if the data block doesn’t exist. NULL may be passed if the value is of no interest.
  • Payload (out): if the specified data block has been found, infData_Get() will reserve memory, copy the payload into that memory portion and return a pointer to it. It is the responsibility of the reader TAP, to free the allocated memory if the payload is not needed anymore. If infData_Get() fails to allocate memory, the return value will still be TRUE but this parameter will receive a NULL pointer. This is also the case if the specified data block doesn’t contain a payload.
  • Return value: TRUE if the specified data block exists.

bool infData_Delete(char *infFileName, char *NameTag);

Removes a specific data block from an inf file.
  • infFileName (in): the name of the file, which needs to be in the current directory.
  • NameTag (in): the unique name of the data block.
  • Return value: TRUE if the specified data block did exist.



bool infData_Set(char *infFileName, char *NameTag, dword PayloadSize, byte *Payload);

Writes a data block into an inf file.
  • infFileName (in): the name of the file, which needs to be in the current directory.
  • NameTag (in): the unique name of the data block. If a data block with the same name already exists, infData_Set() will overwrite the old payload.
  • PayloadSize (in): specifies the size of the payload. It is allowed to specify 0 bytes. In this case a data block with an empty payload will be created.
  • Payload (in): a pointer to the payload. Whenever this parameter is a NULL pointer, a data block without a payload will be written into the inf. If PayloadSize is not 0, a warning will be written to the console.
  • Return value: TRUE if the data block has been created but will be FALSE if Payload == NULL and PayloadSize != 0.

bool infData_GetNameByIndex(char *infFileName, dword NameIndex, char *NameTag);

Retrieves the name of a specific data block.
  • infFileName (in): the name of the file, which needs to be in the current directory.
  • NameIndex (in): specifies the nth data block to be retrieved. Do not cache these indices because the order of the data blocks change with every infData_Set().
  • NameTag (out): receives the name of the specified data block.
  • Return value: TRUE if the specified data block exists.

DeltaMikeCharlie
WebController
WebController
Beiträge: 470
Registriert: Di 7. Mai 2013, 05:11
Wohnort: Australia

AW: Adding data to INF

#18

Beitrag von DeltaMikeCharlie » Di 14. Mai 2013, 22:35

Wow FireBird, that was quick!

I will download your latest library and try it out today.

At the moment, there appears to be no function to update a payload in place. I know that INFplus needs this feature and I can imagine that other TAPs may need it as well.

To address this, how about a function that returns the file offset to the start of the discovered payload block? That way, a TAP could use traditional file IO functions to read and update (but not extend) the data.

I suppose that a TAP could use infData_Delete() and then infData_Set(), but an update function would be more elegant.

This could also be handy when the payload is (or is likely to be) too big to safely fit into RAM in one piece. The function could return the offset and payload size and then the TAP can decide whether to do file IO (read() or mmap()) or simply call infData_Get().

The TAP would have to be careful NOT to append data because the payload may not be at the end of the INF and then PayloadSize would also be wrong. Perhaps you could simply return a pointer that is already mmap()ped (read/write) to the offset, that would also lock in the size and prevent extending the payload.

Benutzeravatar
Twilight
Zauberküchencheflehrling mit extra Butter
Zauberküchencheflehrling mit extra Butter
Beiträge: 64936
Registriert: Fr 9. Dez 2005, 09:17
Receivertyp: 1 x SRP 2100(TMS) TFIR und .1 x SRP 2410 M
Wohnort: Wien Umgebung

AW: Adding data to INF

#19

Beitrag von Twilight » Di 14. Mai 2013, 22:36

thanks :hello:

twilight

Benutzeravatar
FireBird
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Suspekter verdächtiger Zauberküchenchef, TAP & Firmware-Guru
Beiträge: 28969
Registriert: Fr 9. Dez 2005, 09:59
Receivertyp: SRP-2401CI+ TFIR
vu+ Duo 4k
Wohnort: Wien

AW: Adding data to INF

#20

Beitrag von FireBird » Di 14. Mai 2013, 22:43

DeltaMikeCharlie hat geschrieben:At the moment, there appears to be no function to update a payload in place.

If I understand you correctly, infData_Set() will do what you’re looking for. It appends a new payload if it is not contained in the inf, or it replaces the old payload with the new one if the same NameTag already exists.
This could also be handy when the payload is (or is likely to be) too big to safely fit into RAM in one piece.
What are you trying to do with the poor inf? :oa: :lol:

Antworten

Zurück zu „SRP/CRP TAP-Bereich“