I've been tasked with processing a sort of log file that is constantly being updated. These updates are usually fairly slow (maybe 1-2 second) but has bursts where it may add tens to hundreds of lines in a second.
The issue I think I'll have is that the lines have different formats, over 300 different formats so far (still gathering all the possible outputs) and I need to know which of the hundreds of formats match a specific line so I can extract information from that line. This information needs to be displayed in as close to real time as I can get.
Some examples of the patterns of lines are
<fixed text>
- These lines have specific text that never varies. There are lots of these types of lines.
<fixed text> <variable text>
- in this case the line begins with the same information but has something added on the end that is important.
<variable text> <fixed text> <variable text> <fixed text> <variable text> <fixed text>
- In a case line this I need to know all the variable text information.
Now I know I can use regular expressions to get me the information I need (and to identify which type of line it is). I just think running through hundreds of regular expressions looking for one that matches is not the best way to go about this, but can't think of anything else that would work.
I've done some research and have found methods of checking a single string for multiple match strings, but they all use fixed matching strings. I might be able to use these methods to check for the fixed parts of the lines (no two 'patterns' contains the exact same fixed text, well at least that I've seen) then use a regex to pull the information out of the line once it has been identified.
So the question is does anyone have any experience with this type of thing and how did you go about solving it? Speed and accuracy are the important issues here![:)]()
I do want to emphasis that I haven't done any actual testing yet and it may turn out that I'm anticipating an issue that really doesn't exist. I suppose that should be what I do next.
The issue I think I'll have is that the lines have different formats, over 300 different formats so far (still gathering all the possible outputs) and I need to know which of the hundreds of formats match a specific line so I can extract information from that line. This information needs to be displayed in as close to real time as I can get.
Some examples of the patterns of lines are
<fixed text>
- These lines have specific text that never varies. There are lots of these types of lines.
<fixed text> <variable text>
- in this case the line begins with the same information but has something added on the end that is important.
<variable text> <fixed text> <variable text> <fixed text> <variable text> <fixed text>
- In a case line this I need to know all the variable text information.
Now I know I can use regular expressions to get me the information I need (and to identify which type of line it is). I just think running through hundreds of regular expressions looking for one that matches is not the best way to go about this, but can't think of anything else that would work.
I've done some research and have found methods of checking a single string for multiple match strings, but they all use fixed matching strings. I might be able to use these methods to check for the fixed parts of the lines (no two 'patterns' contains the exact same fixed text, well at least that I've seen) then use a regex to pull the information out of the line once it has been identified.
So the question is does anyone have any experience with this type of thing and how did you go about solving it? Speed and accuracy are the important issues here
I do want to emphasis that I haven't done any actual testing yet and it may turn out that I'm anticipating an issue that really doesn't exist. I suppose that should be what I do next.