TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
			Paper
			•
			2404.11912
			•
			Published
				
			•
				
				17
			
we recognize two memory bottlenecks: model weights and KV cache, and the latter gradually bottleneck(s) as context length increases