O debian 32bits é compilado para 386. Apenas alguns pacotes possuem versões otimizadas. (kernel, glibc, ...)
As versões 64bits tem como "máximo divisor comum" o Athlon64, o que já garante algumas otimizações em relação as versões 32bits.
Retirado da página do debian:
http://www.debian.org/ports/amd64/
A complete 64bit userland
The AMD64 port is thoroughly 64bit, allowing the user to benefit from all advantages this architecture has compared to i386:
no memory segmentation into low and high memory
up to 128TiB virtual address space per process (instead of 2GiB)
64TiB physical memory support instead of 4GiB (or 64GiB with the PAE extension)
16 general purpose registers in the CPU instead of 8
gcc defaults to SSE2 math instead of 387 FPU
gcc omits frame-pointers by default at -O2
compilation time optimization uses a common base for AMD64/EM64T instead of legacy i386 cruft
memory pages are not executable by default
Native execution of legacy 32bit binaries is supported by the kernel, and core libraries are provided by the ia32-libs package.