-
Notifications
You must be signed in to change notification settings - Fork 0
/
todo.txt
446 lines (311 loc) · 16.7 KB
/
todo.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
how is it that the Format subclasses are serializable without zero-param constructors?
does Java do some kind of magic with these?
Do serializable classes need to be binary-compatible with JMF in their serialized form? seems like a good idea, FMJ implements this.
Serialization problems: eclipse generates inline fields in some cases (as an optimization) which messes up serialization - makes
it incompatible with the classes in the reference impl jar.
Incompatible classes:
beware of cloning issues (like size).
field == vs equals.
rm FormatUtils tracing and CGUtils when done.
mysteries:
where does Format.copy get called? from clone?
encodingCode? - order of calls in equals is relevant.
static initializers
intersects wierdness
encoding code seems like it would have probs with conflicts.
optimization - all the matching done by the Manager, directly or indirectly, seems slow.
I have to use file:// prefix for files with FMJ, but JMF will find the file without it.
ejmf author emails: ?
Rob Gordon <[email protected]>
Steve Talley <stephen.talley@Central>
http://blogs.sun.com/roller/page/talley/
?
The more I think about JFFMPEG, the more I think that by extending BasicCodec, they possibly use these native buffers which are internal
sun classes. They never create these buffers, which means it is up to the Player to create them. So this codec probably only
uses native buffers when used with the sun player. So if I have my own player, can I use these codecs? Perhaps with a normal buffer.
Now, if my player creates some different kind of native buffer that has the same functionality and contracts as Sun's, I could use their codec
with my JMF and use native buffers. But that would require me to of course provide the same BasicCodec classes to subclass from.
So there appears to be no standard way to use a native buffer (not specified by the main API). If JFFMPEG were rewritten to be
independent of these classes, then it could not use native buffers, since it is the Player that must provide them.
How does the system locate CaptureDevices? There is a manager to get the capture device info. I think a CaptureDevice is generally a DataSource,
so you get the CaptureDeviceInfo from the CaptureDeviceManager, then get the MediaLocator from the CaptureDeviceInfo, then
call Manager.createDataSource with the MediaLocator. The Format has to be get/set through a FormatControl, I suppose.
-------
Generic player
will need a parser (e.g. QuicktimeParser)
renderers and parsers do not appear to be located through the manager.
So a parser, as used by Sun, is an implementation detail. Does not appear to be a javax.media... Parser interface.
thru manager:
protocol (DataSource)
codecs?
content (Handler)
It is confusing because you have the file format (mov)
Then the demux
Then you have the codec(s) (audio, video)
Then you have the renderer - this should be generic.
JFFMPEG has codecs, and demux.
So we need to find a video we can play with JMF/JFFMPEG.
That leaves the actual Player, Renderer, and
content - this is where we need the unknown handler
protocol - the URLDataSource takes care of this for now.
where does the parser fit in?
how is the demux located?
Ok, looking at JFFMPEG, you have to register:
mime types
codecs
demux's
plug ins are managed by the pluginmanager, not the manager.
ok, so we put everything together using the plugin manager.
prefix.content.unknown.Handler
the handler (player) will be given the datasource.
from the datasource, determine content type
(right, we are building a graph)
from the content type, find a ... parser?...
find a demux using content type (demux is parser)
for each stream type coming out of the demux:
using the format, find a codec that has an output format that matches a renderer
may have to create a graph of codecs to do this
Sun's unknown handler is:
com.sun.media.content.unknown.Handler
Ok, so once we use JFFMPEG, then we only need the player, and the renderer. The demux and codec are provided by JFFMPEG
The player only needs to link everything up.
JFFMPEG doesn't work with the stream provided by URL data source. It needs to be seekable.
What is mpeg4t?
FOBS/FMJ has a handler:
com.omnividea.media.content.unknown.Handler
Not really implemented, though.
FOBS problems
- 64-bit linux build
- JavaDecoder/NativeDecoder assume that the size is set when getting input/output formats (JavaDecoder doesn't seem to have this problem in CVS version)
- Parser requires that the DataSource be the FOBS DataSource. Should support other DataSources. FOBS DataSource does not appear to be implemented fully.
- com.omnividea.media.protocol.DataSource has hard-coded:
public String getContentType() {
return "video.ffmpeg";
}
- FOBS handler gets an NPE because videoTrack is not set before syncStart. This appears to be because it is a Processor, a subclass of Player, which has the method configure().
5/22/06
Initial civil integration is there, a lot of rough spots.
correct native civil library needs to be in the library path
TODO: need a browser for capture devices
TODO: support hotplugging of capture devices
5/24/06
Sun appears to have an internal mime manager. Is this what is edited in their registry app?
5/31/06
http://tritonus.org/
Sun's MP3 plugin.
We could extend our sound renderer to handle any format that javasound handles.
But what would the demux do? which demux would we use?
So a the unknown handler would get a standard data source (file), and then look for a demux with the right content
type. So we could create a demux with the content type for WAV, producing raw audio output.
The trick is that javasound uses normal input streams, and jmf uses its own stream interfaces with Buffers.
The demux would use an AudioInputStream.
Actually, we may want the codec to do the work, so that the decoding will work even if WAV audio is embedded in another container format.
There are more than a few gotchas here.
1. if a codec is going to convert between audio formats using java sound, you have the problem that java sound is stream and not buffer oriented.
SO the codec has to take incoming buffers and put them into a fabricated InputStream that returns the buffer data.
A single input buffer may correspond to no output buffer, and the codec is forced to return OUTPUT_NOT_FILLED.
The filter graph needs to be able to handle this. Not sure what the JMF filter graph does. Especially once the input is empty, and
then there is probably a relatively big output buffer awaiting.
2. Where is the right place to support AU/MP3/OGG, when using javasound SPIs? demux? codec? renderer?
3. How are all the buffer sizes to be established - with audio there is no particular buffer size when coming from a file.
I guess what has been confusing me also is whether WAV or AU is a "container" format - what should the demux do with it.
I'm thinking now that the demux needs to provide the raw data via the tracks, but reads enough of it to determine the exact format, and perhaps does not
pass the header info to the caller of track.read.
What is readFrame supposed to do if it encounters an exception, like IOException?
6/29/06:
Some audio files don't playback if JMF is in the classpath.
MP3 - tries to use JFFMPEG.
.au
There is a conflict between com.ibm.media.parser.video.MpegParser and
MP3 with JMF in classpath:
Filter graph:
audio.mpeg
com.ibm.media.parser.video.MpegParser
mpeglayer3, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed, 8000.0 frame rate, FrameSize=16384 bits
net.sourceforge.jffmpeg.AudioDecoder
LINEAR, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed
com.sun.media.renderer.audio.JavaSoundRenderer
WITHOUT:
Filter graph:
audio.mpeg
net.sf.fmj.media.parser.JavaSoundParser
MPEG1L3, 44100.0 Hz, Mono, 38.28125 frame rate
net.sf.fmj.media.codec.JavaSoundCodec
LINEAR, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed, 44100.0 frame rate, FrameSize=16 bits
com.sun.media.renderer.audio.JavaSoundRenderer
AU with JMF in classpath:
Filter graph:
audio.basic
com.sun.media.parser.audio.AuParser
ULAW, 8000.0 Hz, 8-bit, Mono, Signed, FrameSize=8 bits
com.sun.media.renderer.audio.SunAudioRenderer
WITHOUT:
Filter graph:
audio.basic
net.sf.fmj.media.parser.JavaSoundParser
ULAW, 8000.0 Hz, 8-bit, Mono, 8000.0 frame rate, FrameSize=8 bits
net.sf.fmj.media.codec.JavaSoundCodec
LINEAR, 8000.0 Hz, 16-bit, Mono, LittleEndian, Signed, 8000.0 frame rate, FrameSize=16 bits
com.sun.media.renderer.audio.JavaSoundRenderer
10/23/06
One of the things that makes JMF/FMJ programming tricky, is assumptions in the JMF architecture about when formats are known.
There is kind of a chicken-and-egg problem with media: you often need to know the formats before you can build your filter graph, but you often don't know
the formats until you start reading data. That makes it hard to build the filter graph correctly.
JMF's unknown handler calls start on the DataSource during the realization process. It then reads 1 frame, then calls stop. It must do this to get certain info.
Then, when actually starting a clip, it calls start again during prefetch.
getStreams is called after connect, but before start.
FMJ seems to call start twice.
we start the data source during realize
and we start the demux when building the filter graph, also during realize. The RawBufferParser starts the DataSource from there.
The contracts in JMF are very hard to know. Who is supposed to call start on the data source?
Also, what order can things be called on a Track or Stream? For example, can I get the format before/after connect, open?
CIVIL: it would be nice to be able to get the format without reading a frame.
5/30/07
Processor sort of works.
Still hangs at the end running Example5_4Modified
Processes at media speed, not as fast as possible.
July 2 2007
How would something like subtitles be handled in JMF/FMJ? Seems like something that could be a different "track" with a different format.
How about DVD navigation?
How about playlists, CD indexes, DVD indexes? JMF seems to be oriented towards playback of a single multi-track media element, but has no awareness
of structured media like DVDs, with chapters and such.
July 10 2007
TODO:
- RTP transmit/transcode wizard: format selection dlg should allow multiple formats
- transmit JPEG/RTP
- get at bits of native players
see: http://www.onjava.com/pub/a/onjava/2005/06/01/kgpjava_part2.html?page=2
see: http://lists.apple.com/archives/QuickTime-java/2006/Mar/msg00081.html
- use jdshow for lti-civil
Aug 24 2007
What is wrong with FMJ:
- it is awkward to have plugins that span multiple functions, for example a combination DataSource, Demultiplexer and Codec.
There are many cases when wrapping native libraries where it is hard to extract these into their elements, so it would be nice
not to hack it to wrap a larger entity like this.
Actually, for a demux/parser it is not so tricky, since the demux just produces the decoded data.
However, for a multiplexer this will be more complex, since generally the calling code tells the filter graph
what the input to the muliplexer should be, for example mpeg video.
- there needs to be a general clarification of vague parts of the spec. There are TODOs littered
around the code that point these out. For example:
- when a failure occurs in opening/starting a demux, who is responsible for the cleanup?
- what happens when both the EOM and discard bits are set in a buffer. Does that discard the EOM?
Directshow capture:
http://msdn2.microsoft.com/en-us/library/ms867162.aspx
Sep 11 2007
To get FMJ to work with JMF in the classpath (JMF first), there are a few issues:
1. the FMJ registry defaults are not there - done
2. relative file URLs not supported - done
3. safexmas.mov uses a heavyweight component that paints over the url drop down
Not a problem with FMJ, but noticed that playing something like internet radio doesn't work well with JMF, because it is sending lots of caching progress events, which never seem to stop...
TODO: Files to work on with FMJ ahead in classpath:
streaming ogg/mp3
TODO: getting 403 from
//prefs.recentUrls.add("http://upload.wikimedia.org/wikipedia/commons/d/d0/Apollo_15_liftoff_from_inside_LM.ogg"); // Apollo 15 movie from wikimedia
Sep 12 2007
Running with FMJ ahead in classpath:
file:///home/ken/Desktop/4harmonic.mpg
No audio plays, jmf.log:
## Here's the completed flow graph:
net.sf.fmj.ffmpeg_java.FFMPEGParser@44323274
connects to: net.sf.fmj.media.renderer.video.SimpleAWTRenderer@f5d8d75
format: RGB, 320x240, FrameRate=25.0, 24-bit, Masks=1:2:3, PixelStride=3, LineStride=960
net.sf.fmj.ffmpeg_java.FFMPEGParser@44323274
connects to: com.sun.media.renderer.audio.JavaSoundRenderer@57f5b4d1
format: LINEAR, 32000.0 Hz, 16-bit, Mono
$$ Profile: graph building: 64 ms
$$ Profile: realize, post graph building: 16 ms
!! DataLine not supported: LINEAR, 32000.0 Hz, 16-bit, Mono
XX Unable to handle format: LINEAR, 32000.0 Hz, 16-bit, Mono
This works if we remove all JMF plugins/packages from registry.
It is starting to become clear that some of the JMF classes have some serious limitations, and that ultimately, we
only want to use the ones that have no FMJ counterpart. Need to divide JMF plugins into good bad and ugly...
TODO: http://www.archive.org/download/bunny_goes_down_2006/bunny_goes_down_2006_64kb.mp4 plays back slowly.
Something to try:
rtsp://ia301238.us.archive.org:554/0/items/DIA40/DIA40_64kb.mp4
http://www.archive.org/download/DIA40/DIA40_64kb.mp4
slowly - constanly reopening http stream, to see, promote URLDataSource: logger.finer("Full content type: " + conn.getContentType());
Plays, timing is bad:
http://ia300129.us.archive.org/3/items/salmat/salmat.flv
TODO: safexmas.mov plays twice as fast as it should.
Sep 25 2007
Using AVTransmit2:
JMF's stack with FMJ: RawBufferSourceStream read never gets called
JMF appears to call setInputFormat on the mux for lots of possible input formats, to determine available input formats.
TODO:
playback of last bit of au twice
preview wizard
receive wizard?
RTP reception doesnt' work second time. Is something not cleaning up?
loop has problems
type in location, then click load, then it gives 2 errors if error.
preview wizard
offer alaw and speex
jpeg packetizer
data sink for Internet radio streaming.
m3u/playlists
cd audio
auto-loop in fmj studio
support /dev/video7
ip cameras
ip server
player:
transcode wizard
maintain aspect ratio
full screen
snapshot
plugin viewer
capture control
transmission stats
controls (frame grabbing controls)
wrap native codecs for processing
synchronization
implement java codecs for easy ones (jpeg, etc).
controls, graphical controls
details of format in capture urls
native buffers
Sep 27 2007
It does appear that FMJ's unknown handler will break data into chunks before feeding it to the renderer. It will take a big 1k buffer and
pass in offset 0 len 500 then offset 500 len 500, or something like that. Observed when sending 1k RTP audio packets, and seeing how
it calls the java sound renderer.
Sep 29 2007
When FMJ requests a URL, here are the params:
INFO: HDR: 'User-Agent' = 'Java/1.5.0_07'
INFO: HDR: 'Accept' = 'text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2'
INFO: HDR: 'Content-type' = 'application/x-www-form-urlencoded'
INFO: HDR: 'Host' = 'localhost:8090'
INFO: HDR: 'Connection' = 'keep-alive'
Oct 8 2007
JMF's reflection-based loading mechanism is suboptimal for applets, because it then tries to load lots of nonexistant classes from the web server, that are not in the jar.
Oct 10 2007
javasound demuxes need to enforce their specific formats
still need to deal with the fact that most sound demuxes need to rewrite their header at the end
need a way of merging formats so that if a less specific one is set, the merged format is the result.
Oct 11 2007
test FmjTranscode with JMF in classpath...
Oct 21 2007
Top issues:
suboptimal filter graphs
unspecified formats in filter graphs (negotiation with mux)
civil crash
audio mux hangs
playback restart freeze
Oct 22 2007
review of JMF javadoc, things that FMJ doesn't really use yet:
prefetching
CachingControl,ExtendedCachingControl
should a codec block rather than return flags that call it again?
Drainable?
Prefetchable
DataSinkListener, DataSinkEvent events
RateConfiguration,RateConfigureable
Mux:
"If a Multiplexer implements the Clock interface, it can be used by a Player as the master time base for synchronization. In this case, the Multiplexer should update the media time and time-base time as it processes the media."
Oct 29 2007
suboptimal filter graphs fixed.
Still need to deal with things like image scaling conversions, and the full set of supported mux input formats.
Nov 11 2007
Need YUV/RGB conversions before implementing ffmpeg encoding
Mar 10 2008
Use MemoryImageSource for more efficiency.