What languages do not use the 'numerical alphabet' that we are all so used to: 1, 2, 3, etc.?

The Celtic one.

https://en.wikipedia.org/wiki/Vigesimal (this says this and the Mayan used base 20- satguru)

There is also the Maya numerical system, which is different from ours.

https://en.wikipedia.org/wiki/Maya_numerals

The Piraha culture in the Amazonas does not have numbers at all.

http://www.spiegel.de/international/spiegel/0,1518,414291,00.html

Most use the Arabic system... i.e. 1,2,3 etc. This is known as the decimal, or to base ten, system, and there are many others: binary, quaternaty, octal, hexadecimal - all of which use some or all of the numbers 1,2,3,4,5,6,7,8,9,0 and A,B,C,D,E,F.

Then there is Roman numerals: I,V,X,L,C,D,M.
There are also others on wikipedia - take a browse!
https://en.wikipedia.org/wiki/Numeral_system

I'll just paste this in from my link above.

"The system of Hebrew numerals is a quasi-decimal alphabetic numeral system using the letters of the Hebrew alphabet.

"In this system, there is no notation for zero, and the numeric values for individual letters are added together. Each unit (1, 2, ..., 9) is assigned a separate letter, each tens (10, 20, ..., 90) a separate letter, and each hundreds (100, 200, ..., 900) a separate letter."

As far as I know, of the ancient languages only Chinese had distinct symbols for numbers.

https://en.wikipedia.org/wiki/Rod_numerals

Notice the box entitled "Numeral systems by culture" to the right on the page above for more details. "Arabic" numerals were devised in India by about 500 CE.
https://en.wikipedia.org/wiki/Arabic_numerals