shivers@ai.mit.edu (Olin Shivers) writes:
> Could you post the most trivial possible example that manifests your problem?
Sure.
I even started with a freshly-built scsh distribution on a
normal (if very busy and very slow) SunOS 4.1.4 host, just
to eliminate any possible variables related to my
somewhat more customized environment.
> (define-syntax stream-cons
(syntax-rules ()
((stream-cons head tail)
(cons head (delay tail)))))
> (define stream-car car)
> (define (stream-cdr s) (force (cdr s)))
> (define the-empty-stream '<the-empty-stream>)
> (define the-null-stream (stream-cons the-empty-stream the-null-stream))
> (define (file->stream f)
(let ((p (open-input-file f)))
(letrec ((build-input-stream
(lambda ()
(let ((ch (read-char p)))
(if (eof-object? ch)
(begin
(close-input-port p)
the-empty-stream)
(stream-cons ch (build-input-stream)))))))
(build-input-stream))))
> (define (stream-null? s) (if (eq? (stream-car s) the-empty-stream) #t #f))
> (define (stream->file f s)
(let ((p (open-output-file f)))
(letrec ((write-stream
(lambda (stream)
(if (not (stream-null? stream))
(begin (write-char (stream-car stream) p)
(write-stream (stream-cdr stream)))))))
(write-stream s)
(close-output-port p))))
> (cwd)
"/home/smd"
> ,time (stream->file "X" (file->stream ".emacs"))
Run time: 33.30 seconds
#t
> (run (cmp ".emacs" "X"))
0
;; this was a SMALL file. now let's do it with a HUGE file.
;; the theory here is that we recycle garbage at the rate
;; that we create cons cells, so running very large
;; streams based things should work
> (run (ls -l ".emacs"))
-rw------- 1 smd 14752 Aug 5 20:11 .emacs
0
> (run (ls -l "/var/log/httpd/access_log"))
-rw-r--r-- 1 root 32021078 Aug 21 14:53 /var/log/httpd/access_log
0
> ,time (stream->file "Y" (file->stream "/var/log/httpd/access_log"))
Interrupt: memory-shortage
1>
[parenthetical comment:
: cesium.clock.org ; ls -l X Y
-rw------- 1 smd 14752 Aug 21 14:51 X
-rw------- 1 smd 40960 Aug 21 14:57 Y
]
Interrupt: memory-shortage
1> ,collect
Interrupt: memory-shortage
2> ,reset
> ,collect
Before: 81088 words free in semispace
After: 682561 words free in semispace
[I shall spare you the repeat without the ,time. The
failure mode is exactly the same.]
Another quickie example:
> (define (stream-map proc stream)
(if (stream-null? stream)
the-null-stream
(stream-cons
(proc (stream-car stream))
(stream-map proc (stream-cdr stream)))))
> (define (m)
(letrec ((positive-integers
(stream-cons 1 (stream-map (lambda (a) (+ a 1)) positive-integers))))
positive-integers))
> (define (stream-nth stream n)
(if (stream-null? stream)
the-empty-stream
(if (= n 0)
stream
(stream-nth (stream-cdr stream) (- n 1)))))
> (stream-nth (m) 100)
'(101 . #{Procedure 299 (unnamed in make-promise)})
> (stream-nth (m) 1000)
'(1001 . #{Procedure 299 (unnamed in make-promise)}
> (stream-nth (m) 10000)
'(10001 . #{Procedure 299 (unnamed in make-promise)})
> (stream-nth (m) 100000)
Interrupt: memory-shortage
1> ,reset
> ,collect
Before: 97723 words free in semispace
After: 552531 words free in semispace
> (stream-nth (m) 100000)
Interrupt: memory-shortage
1> ,debug
'#{Continuation (pc 16) signal}
[0] 'interrupt
[1] '(2 7)
inspect: d
'#{Continuation (pc 0) 0}
[0] 7
[1] 1
[2] 0
[3] '#{Template 298 make-promise}
[4] '#{Procedure 298 make-promise}
[5] '#{Procedure 10193 (unnamed in stream-map)}
inspect: d
'#{Continuation (pc 53) stream-map}
Waiting for (make-promise (lambda () (stream-map proc #)))
in (cons (proc (stream-car stream)) ^^^)
[0] 36509
[1: proc] '#{Procedure 10263 (unnamed in unnamed in m)}
[2: stream] '(36508 . #{Procedure 299 #})
inspect: d
'#{Continuation (pc 53) (unnamed in make-promise)}
[0] #f
[1] #t
[2] '#{Procedure 10193 (unnamed in stream-map)}
inspect: d
'#{Continuation (pc 52) stream-nth}
Waiting for (stream-cdr stream)
in (stream-nth ^^^ (- n 1))
[0: stream] '(36508 . #{Procedure 299 #})
[1: n] 63493
inspect: d
'#{Continuation (pc 21) evaluate-and-select}
[0] '#{Procedure 5463 (unnamed in evaluate-and-select)}
[1] '(stream-nth (m) 100000)
[2] '#{Package 113 user}
inspect: d
'#{Continuation (pc 13) (unnamed in dynamic-wind)}
[0] '#{Procedure 230 list}
[1] '#{Procedure 5512 (unnamed in execute-command)}
[2] '#{Procedure 5513 (unnamed in execute-command)}
[3] '#{Procedure 5486 run-sentinels}
inspect: d
'#{Continuation (pc 51) with-dynamic-env}
[0] '#{Procedure 480 (unnamed in with-dynamic-env)}
[1] '((# . #f) (# #) (# # # #) (# . #) (# . #) ---)
[2] #f
[3] #f
inspect: d
'#{Continuation (pc 91) dynamic-wind}
[0] '#{Procedure 5512 (unnamed in execute-command)}
[1] '#{Procedure 5513 (unnamed in execute-command)}
[2] '#{Procedure 5486 run-sentinels}
inspect: d
'#{Continuation (pc 23) showing-focus-object}
[0] '((10001 . #))
[1] '#{Procedure 5429 (unnamed in loop in unnamed ---)}
inspect: d
'#{Continuation (pc 51) (loop in unnamed in unnamed ---)}
[0] '(run (stream-nth # 100000))
[1] '#{Procedure 5428 (loop in unnamed in unnamed ---)}
[2] '#{Procedure 5438 (unnamed in unnamed in abort-to-command-level)}
[3] '#{Command-level}
inspect: d
'#{Continuation (pc 51) with-dynamic-env}
[0] '#{Procedure 480 (unnamed in with-dynamic-env)}
[1] '((# # # #) (# . #) (# . #) (#) (# . #) ---)
[2] #f
[3] #f
inspect: d
'#{Continuation (pc 51) with-dynamic-env}
[0] '#{Procedure 480 (unnamed in with-dynamic-env)}
[1] '((# . #) (# . #) (#) (# . #) (# # #) ---)
[2] #f
[3] #f
inspect: d
'#{Continuation (pc 51) with-dynamic-env}
Interrupt: memory-shortage
2> ^D
Back to
Interrupt: memory-shortage
1> ,collect
Interrupt: memory-shortage
2> ,reset
> ,collect
Before: 73888 words free in semispace
After: 682526 words free in semispace
>
[maybe of interest -- gc finding something that points to garbage?]
> ,collect
...
After: 422500 words free in semispace
> (stream-nth (m) 25000)
'(25001 . #{Procedure 299 (unnamed in make-promise)})
> ,collect
Before: 62376 words free in semispace
After: 357500 words free in semispace
> (stream-nth (m) 25000)
Interrupt: memory-shortage
1> ,reset
> ,collect
Before: 50104 words free in semispace
After: 357500 words free in semispace
> (stream-nth (m) 10000)
'(10001 . #{Procedure 299 (unnamed in make-promise)})
> ,collect
Before: 150775 words free in semispace
After: 552500 words free in semispace
> (stream-nth (m) 10000)
'(10001 . #{Procedure 299 (unnamed in make-promise)})
> ,collect
Before: 345772 words free in semispace
After: 552500 words free in semispace
;; same again, now let's try 20000
> (stream-nth (m) 20000)
'(20001 . #{Procedure 299 (unnamed in make-promise)})
> ,collect
Before: 145296 words free in semispace
After: 422500 words free in semispace
;; why don't I get back 552500 words free?
> (stream-nth (m) 20000)
'(20001 . #{Procedure 299 (unnamed in make-promise)})
> ,collect
Before: 15296 words free in semispace
After: 422500 words free in semispace
> (stream-nth (m) 20000)
'(20001 . #{Procedure 299 (unnamed in make-promise)})
> ,collect
Before: 15299 words free in semispace
After: 422500 words free in semispace
;; but not if I go and do 10000 again...
> (stream-nth (m) 10000)
'(10001 . #{Procedure 299 (unnamed in make-promise)})
> ,collect
Before: 215775 words free in semispace
After: 552500 words free in semispace
Why is there garbage that isn't collected in this case?
Moreover, is this related to running out of space
using the stream stuff above?
Also, for fun, I did this:
;; run 20000, do collect
> ,collect
Before: 413051 words free in semispace
After: 422486 words free in semispace
> (display 'a)
a> ,collect
Before: 416773 words free in semispace
After: 422478 words free in semispace
> (display 'a)
a> ,collect
Before: 416773 words free in semispace
After: 422478 words free in semispace
> (m)
'(1 . #{Procedure 299 (unnamed in make-promise)})
> ,collect
Before: 417219 words free in semispace
After: 682485 words free in semispace
eq? eqv? and (equal? m m) give me #f.
why does invoking a subsequent (m) lead
to the release of garbage created by
processing a stream created by a previous (m)?
Unfortunately, this only happens in the top-level repl
after (stream-nth (m) ...) has happened; if I run (m) in
debug space or in stream-nth, the created garbage doesn't
go away.
Sean.
|