This article is somewhat technical. If you just want to listen to a chip tune on WebMSX, maybe go for part 2 instead.
In previous articles I explored the YM2151 and the VGM file format. In this article, we’ll go back a generation and listen to some tunes written for the DSG (doorbell sound generator) PSG (programmable sound generator, i.e., the General Instruments AY-3-8910, or compatibly, Yamaha’s YM2149). PSG files (and particularly ASC files) are mainly used for ZX Spectrum chip tunes (I think), but the MSX has the same sound chip so why not play some chip tunes on the MSX?
Well, before we spend time working on something just slightly above PC beeper music… are there even any decent PSG tunes? Well, I’ve found at least one that like, “Popsa 2”, as is included in the below mix (scroll down a bit) on YouTube for example, and some of the commenters on this video seem to like “Illusion”.
Unfortunately, it doesn’t work in WebMSX (after 20 seconds or so). But it works in all three (NTSC) openMSX machines I bothered to test with, and it also works on my real MSX1 (Hitachi MB-H2). To get it to run in WebMSX, you have to “Set ROM format” -> “KonamiSCC”, but even then it’ll crash after a few minutes (vs. 20 seconds for e.g. ASCII8). For some reason it doesn’t let me choose “Normal”. I’m quite sure it would work with that setting if it were available. :p I’ll look into the matter at some point, probably. Looks like WebMSX will require a patch to work. Patch is submitted and will probably make it into the next version.
Caution: writing to certain PSG registers is unsafe on certain MSX machines. I don’t think my code writes to these registers, but I didn’t make 100% sure. (However, openMSX gives you a warning when it notices unsafe writes, and I didn’t get a warning.)
“Popsa 2” was made in a program called ASC Sound Master. The “.asc” file can be downloaded here: https://zxart.ee/eng/authors/d/dreamer/popsa-2/. These .asc files are pretty small. They can be converted to PSG using ZXTune (https://bitbucket.org/zxtune/zxtune/) (and from PSG they can easily be converted to e.g. VGM, see bottom of this post), but the resulting files are too large to fit on a regular MSX1 cartridge.
ZXTune compilation and conversion:
git clone https://bitbucket.org/zxtune/zxtune.git
cd zxtune
make platform=linux system.zlib=1 -C apps/zxtune123/ -j4
bin/linux/release/zxtune123 --convert mode=psg,filename=foo.psg -- Dreamer\ -\ POPSA-2\ \(1994\).ascThe original .asc file is 3720 bytes. The resulting .psg is 129028 bytes. If you convert that to VGM, the resulting size is 187342 bytes.
The PSG file format
The PSG file format is very similar in concept to the VGM file format, except that only one chip is supported, the PSG. It seems it’s primarily used for ZX Spectrum chip tunes. As only one chip is supported, you don’t need the “command byte” that indicates what chip is to be written to. So you only have pairs of “register address” and “register value to write”.
There’s also a header in the first 16 bytes. The first three bytes are “PSG”, dunno about the rest.
The PSG only has 16 (IIRC) registers, and some of those aren’t even relevant for sound. In other words, the registers 0x10 to 0xff don’t exist and the designers of this file format used that opportunity to fit in a “wait” command at 0xff (one raster scan, so 1/50s or 1/60s depending on whether the system is PAL or NTSC). There’s also a command that waits multiple raster intervals, 0xfe, and a command that ends the tune, 0xfd. Ignoring the header, here are the first few bytes of the Popsa 2 PSG file:
ff
00 41
01 05
02 0b
03 01
04 e0
05 00
06 1f
07 31
08 0f
09 0f
0a 0a
ff
02 e0
03 00
07 38
09 0d
0a 0c
ff
...All this means: wait 1 raster interval, then write to registers 00 through 0a with values 41, 05, 0b, 01, …, respectively, wait 1 raster interval, write to registers 02, 03, 07, 09, 0a, with values e0, 00, 38, 0d, 0c, respectively, wait 1 raster interval. (As you can see the 0xff command doesn’t take any parameters.)
Now that we know mostly how this file format works, it’s time to think about how to fit roughly 126 KB of data into my 48 KB cartridge. We could easily use an off-the-shelf compression library, but where’s the fun in that? That’s like… modern programming, ew.
We’ll invent another command for PSG, 0xfc, which takes a two-byte parameter that tells it to jump back somewhere (for a while, and then returns to its original location). We also need to write a program that identifies repetitive sections in the music (of which there are plenty). The former is pretty easy, so let’s talk about the latter program first.
- Compute MD5 sums of a 100-byte window for every byte in the file. So we end up with 129028-100=128928 MD5 sums. Easy and fast on modern hardware. See code snippet below.
- Check if we even have repeated chunks, e.g. by executing: md5sum chunks/* | awk ‘{print $1}’ | sort -n | uniq -c
- We may want to check a couple other window sizes to see if we can get better results. A lower window size means we’ll find more repetition, but we need 3 bytes to encode a jump in our PSG file.
- Re-assemble PSG file using a quick-and-dirty and probably somewhat buggy script. (See below.)
The resulting data length is 42158 bytes for the Popsa 2 song.
For task (1) we first convert the PSG file into hex, and later into tokens:
xxd -p Dreamer\ -\ POPSA-2\ \(1994\).psg | sed -r -e 's/(..)/\1 /g' | tr -d '\n' > Dreamer\ -\ POPSA-2\ \(1994\).psg.hex
# Then remove 16-byte header using a standard text editorcat Dreamer\ -\ POPSA-2\ \(1994\).psg.hex | perl -e '$_ = <STDIN>; while (s/(..) //) { $cmd = $1; if ($cmd eq "ff" or $cmd eq "fd") { print("$cmd\n") } else { s/(..) //; $param = $1; print("$cmd $param\n") } }' > tokensThen divide the tokens into chunks using the below script, divide_tokens_into_chunks.sh:
#!/bin/bash
N=100 # sliding window length
mkdir -p chunks_N$N
line_count=$(cat tokens | wc -l)
for ((i=0; i<$((line_count-N)); i++)); do
tail -n +$i tokens | head -n $N > chunks_N$N/chunk_$i
done
rm chunks_N$N/chunk_0 # same as chunk_1You know, looking back at this code for the first time in a while, I see there’s a nice off-by-1 error and a nice rm command to fix half of the problem. But the great thing about this being a hobby is that I don’t need to care. :)
Next, we have a Perl script that creates our PSG file. It needs some help though, so we do this first:
find chunks_N100/ | xargs md5sum > chunks_N100_md5sums(We can’t do md5sum chunks_N100/* because that expands to a tad too many arguments in our case. xargs automatically cuts down the number of arguments to a more reasonable value.) This is the main program. Usage: ./compress_aggressive_but_convert_to_psg.pl < chunks_N100_md5sums > foo.psg
#!/usr/bin/perl
# dependencies:
# chunks_N$N/ (directory)
# chunks_N$10_md5sums (file) # example generation: find chunks_N10/ | xargs md5sum > chunks_N10_md5sums
use strict;
use warnings;
use feature "switch";
my $N = 100;
my $md5s = {};
my @chunks;
my $md5;
my $file;
my $debug_logged = 0;
my $lines = [];
my $current_output_byte_number = 0;
for (my $chunk_number = 0; <>; $chunk_number++) {
/([a-z0-9]+)\s+([a-zA-Z0-9_\/]+)/;
$md5 = $1;
$file = $2;
if (exists $md5s->{$md5}) {
# can't call chunks that already contain a call because that call would take us beyond the N token window that we can see from where we are
# that means it's likely we'd generate wrong code
# so we'll just move on and maybe we'll find a nicer block
my $target_chunk_number = $md5s->{$md5}->{chunk_number};
my $concatted_chunks = join('', @chunks[max(0, $target_chunk_number-$N)..min($#chunks, $target_chunk_number+$N)]);
if (($concatted_chunks =~ /; call/) or # NOTE "call wait_for_raster" is allowed
($chunk_number - $target_chunk_number < $N)) {
# 1) can't convert due to existing call; nothing to be done here, or
# 2) we can't call something right behind us
# DANGER let's head back to the non-exists path
goto NON_EXIST_PATH;
} else {
if (!$md5s->{$md5}->{converted_to_call}) {
convert_to_callable_sub($target_chunk_number);
$md5s->{$md5}->{converted_to_call} = 1;
}
my $output_byte_number_high = int($md5s->{$md5}->{output_byte_number} / 256);
my $output_byte_number_low = $md5s->{$md5}->{output_byte_number} % 256;
$chunks[$chunk_number] = sprintf("fc %02x %02x ; call " . $md5s->{$md5}->{output_byte_number} . " ($md5)\n", $output_byte_number_high, $output_byte_number_low);
$current_output_byte_number += 3;
# skip next N-1 rows
for (0..$N-1) {
my $foo = <>;
$chunk_number++;
$chunks[$chunk_number] = "";
}
}
} else {
$md5s->{$md5} = {};
$md5s->{$md5}->{chunk_number} = $chunk_number;
$md5s->{$md5}->{converted_to_call} = 0;
NON_EXIST_PATH:
open my $fh, '<', $file or die "Can't open \"$file\": $!";
my $token = <$fh>;
close $fh;
my $asm = convert_to_asm($token);
$md5s->{$md5}->{output_byte_number} = $current_output_byte_number;
$current_output_byte_number += (scalar(split(" ", $asm)));
$chunks[$chunk_number] = $asm;
}
}
print foreach @chunks;
print "infloop:
jr infloop\n";
# no changes needed
sub convert_to_callable_sub($) {
my $block_number = shift;
}
# don't actually do anything here
sub convert_to_asm($) {
my $string = shift;
return "$string";
}
sub min($$) {
my ($a, $b) = @_;
return $a if ($a < $b);
return $b;
}
sub max($$) {
my ($a, $b) = @_;
return $a if ($a > $b);
return $b;
}The output of this program is in hex. Now we just need some assembly code to read the data and put it into the PSG registers. Here’s the core part:
ld hl,psg_begin
main_loop:
ld a,(hl)
cp 0xff
jr z,wait
cp 0xfe
jr z,wait_n_times
cp 0xfd
jr z,end
cp 0xfc
jr z,jump
jr register_write
inc_loop:
inc hl
jr loop
wait:
call wait_for_raster
jr inc_loop
register_write:
ld a,(hl)
out (0xa0),a
inc hl
ld a,(hl)
out (0xa1),a
jr inc_loop
wait_for_raster:
in a,(0x99)
and 128
cp 128
jr nz,wait_for_raster
ret
psg_begin:
include "foo.psg"
ds 010000h-$ ; fill rest with 0sUnderstanding the above should help understanding the full implementation. (The above doesn’t include the code for the 0xfe, 0xfd, and 0xfc commands.) Note that we can’t use the above wait_for_raster on NTSC machines because the tune assumes 50 Hz. So we’ll instead emulate the 50 Hz interval using a busy loop.
For 0xfd (end of song), we just enter an infinite loop. For 0xfe, we just call wait_for_raster multiple times. For 0xfc, we need to store where we left off, then set hl to the address in the parameter, then execute exactly 100 main loop runs, then set hl back to its previous address and continue as normal.
Here’s the code, which also includes some VRAM writes to visualize the music a little bit. Does it look good? Eh, I dunno. It was an experiment. I changed the registers to be displayed because some registers don’t see updates very often. The overall visuals are a bit noisy, but there is one section that looks good in my opinion, and it’s also the section that I like best in the tune, right at the end. You can clearly see one of the registers changing right in sync with the doorbell sound. (It looks even more in sync in openMSX.)
N: equ 100
org 4000H
db "AB"
dw entry_point
db 00,00,00,00,00,00,00,00,00,00,00,00
SetVdpWrite: macro high low ; from http://map.grauw.nl/articles/vdp_tut.php
ld a,low
out (0x99),a
ld a,high
add 0x40
out (0x99),a
endm
vpoke: macro value
ld a,value
out (0x98),a
endm
entry_point:
; copy cart rom (c000-f000) to ram
in a,(0a8h)
and 11000000b ; we want to know which slot is RAM, and AFAIK RAM should be mapped in at 0xc000-0xffff.
ld c,a ; save value for later
in a,(0a8h)
and 00001100b ; we are executing from cartridge ROM at 0x4000~0x7fff, so the 2-bit value for this region is known correct. we just have to make the slots above this one the same value.
ld b,a ; save a
rla ; << 1 (now have 000xx000b)
rla ; << 1 (now have 00xx0000b)
or b ; | saved b (now have 00xxxx00b)
rla ; << 1 (now have 0xxxx000b)
rla ; << 1 (now have xxxx0000b)
or b ; | saved b (now have xxxxxx00b)
; ld a,01010100b ; set pages 0: rom 1: rom 2: cart 3: cart
out (0a8h),a
copy_c000_f000:
ld hl,0c000h ; start at c000
copy_c000_f000_loop:
ld a,(hl) ; read from ROM address (hl)
ld d,a
in a,(0a8h)
ld b,a ; store original value
and 00111111b ; only keep settings for lower three slots
or c ; add in setting for top slot (saved earlier)
; ld a,011010100b
out (0a8h),a ; set port
ld (hl),d ; store value read from ROM address (hl) to RAM address (also hl of course)
ld a,b ; load a with original value
out (0a8h),a ; set port back
inc hl
ld a,h
cp 0f0h
jp z,other_init ; done with this copy
jp copy_c000_f000_loop
; entry_point:
; ld a,0xd4
; out (0xa8),a ; set slots
other_init:
; set ports to bios:cart:cart:ram
in a,(0a8h)
and 00111111b ; only keep settings for lower three slots
or c ; add in setting for top slot (saved earlier)
out (0a8h),a ; set port
; set colors
ld a,011110000b ; set data to be written into register (white on black)
out (099h),a
ld a,010000111b ; set register number (7)
out (099h),a
SetVdpWrite 0x20 0x05
vpoke 0x0f ; set white on black for some part of the screen
vpoke 0x0f ; set white on black for some other part of the screen
video_init:
; put chars /0123456789 into 0x1800-0x1AFF
SetVdpWrite 0x18 0x00
ld b,64 ; 64 chars
video_loop_1:
vpoke 0x2f
djnz video_loop_1
ld b,64 ; 64 chars
video_loop_2:
vpoke 0x2f
djnz video_loop_2
ld b,64 ; 64 chars
video_loop_3:
vpoke 0x33
djnz video_loop_3
ld b,64 ; 64 chars
video_loop_4:
vpoke 0x33
djnz video_loop_4
ld b,64 ; 64 chars
video_loop_5:
vpoke 0x36
djnz video_loop_5
ld b,64 ; 64 chars
video_loop_6:
vpoke 0x36
djnz video_loop_6
ld b,64 ; 64 chars
video_loop_7:
vpoke 0x31
djnz video_loop_7
ld b,64 ; 64 chars
video_loop_8:
vpoke 0x31
djnz video_loop_8
ld b,64 ; 64 chars
video_loop_9:
vpoke 0x35
djnz video_loop_9
ld b,64 ; 64 chars
video_loop_10:
vpoke 0x35
djnz video_loop_10
ld b,64 ; 64 chars
video_loop_11:
vpoke 0x37
djnz video_loop_11
ld b,64 ; 64 chars
video_loop_12:
vpoke 0x37
djnz video_loop_12
ld b,64 ; 64 chars
ld b,0 ; flag to indicate whether we are jumping around at the moment (0 means we aren't) (NOTE: nested jumping isn't supported)
ld c,0xa0 ; first PSG port
ld hl,psg_begin
jr main_loop
loop:
ld a,b
cp 0
jr z,main_loop ; b isn't set so just head back to the loop
pop af
dec a
cp -1
jr z,restore_hl
push af ; don't need this on the stack if we go to restore_hl, so place it after the jump
jr main_loop
restore_hl:
ld b,0 ; unset flag
pop hl
inc hl
; and continue executing into loop
main_loop:
ld a,(hl)
cp 0xff
jr z,wait
cp 0xfe
jr z,wait_n_times
cp 0xfd
jr z,end
cp 0xfc
jr z,jump
jr register_write
inc_loop:
inc hl
jr loop
wait:
call wait_for_raster_50hz_emu
jr inc_loop
wait_n_times: ; safe to assume that param isn't 0
push bc
inc hl
ld b,(hl)
wait_n_times_loop:
call wait_for_raster_50hz_emu
djnz wait_n_times_loop
pop bc
jr inc_loop
end:
jr end ; infinite loop
jump:
inc hl
ld d,(hl)
inc hl
ld e,(hl)
push hl
ld b,1 ; signal that we're calling a previous segment
ld a,N ; we want to execute N instructions before going back to where we left off
push af
ld hl,psg_begin
add hl,de
jr loop
register_write:
ld a,(hl)
out (c),a
; really we only need ld a,(hl) and out (0xa1),a, but let's poke around in the VRAM to make this program slightly less boring
; we'll modify the tile definitions of characters /, 0, ..., 9 (8 bytes each starting at 0x178) and just put in the same value we're writing to the PSG register
or a ; clear carry flag to make rla behave
; a = a*8 for vram write address
rla ; *2
rla ; *2 (*2*2 == *4)
rla ; *2 (*2*2*2 == *8)
ld d,a ; vram write address
inc hl
ld a,(hl)
ld e,a ; vram write value
out (0xa1),a
ld a,0x78
add a,d ; vram address low byte is 0x78 + (psg register)*8
; color change code currently commented out because it's not very pleasant to look at
; ; let's also change some colors when register 5 is written to, which doesn't appear to happen very often
; ; for register 5 a is 5*8 + 0x78 = 0xa0
; cp 0xa0
; jr nz,skip_color_change
; ld d,a
; ld a,e
; out (099h),a
; ld a,010000111b ; set register number (7)
; out (099h),a
; ld a,d
skip_color_change:
SetVdpWrite 1 a ; vram address high byte is 1 (full address: 0x178)
vpoke e
vpoke e
vpoke e
vpoke e
vpoke e
vpoke e
vpoke e
vpoke e
jr inc_loop
wait_for_raster:
in a,(0x99)
and 128
cp 128
jr nz,wait_for_raster
ret
wait_for_raster_50hz_emu:
; CPU clock is 3579545 Hz
; decrement and loop routine takes 36 instructions per loop run (wait_for_raster_50hz_emu_loop up to (not including) low_0)
; (https://www.overtakenbyevents.com/tstates/)
; want routine to finish in 1/50 or a second, so:
; 3579545/50/36=1988.636111111111, let's very scientifically, er, let's throw out that whole calculation and say 1650 because we have overhead and I have experimentally determined that to sound close enough to the original :p
; our overhead varies depending on code path. some rhythm problems are audible, but not _too_ terrible
ld de,1650
wait_for_raster_50hz_emu_loop:
dec de
ld a,e
cp 0
jr z,low_0
jr wait_for_raster_50hz_emu_loop
low_0:
ld a,d
cp 0
jr z,high_low_0
jr wait_for_raster_50hz_emu_loop
high_low_0:
ret
psg_begin:
include "foo.psg"
ds 010000h-$Compiles with z80asm. Other assemblers might need some tweaks.
Bonus: converting PSG files to VGM
This is implemented in straight-forward C.
Compilation: cc -o psg2vgm psg2vgm.c
Execution: ./psg2vgm Dreamer\ -\ POPSA-2\ \(1994\).psg | xxd -r -p > foo.vgm
#include <stdio.h>
#include <stdlib.h>
int try_fgetc(FILE *file)
{
int param;
if ((param = fgetc(file)) == EOF) {
fprintf(stderr, "Unexpected EOF");
exit(1);
}
return param;
}
int main(int argc, char **argv)
{
FILE *file = fopen(argv[1], "r");
int c, param;
int i;
// header with 3579545/2 = 1789772 Hz clock rate for AY8910, fake value for length (should be long enough for anyone)
printf("56 67 6d 20 f0 f0 f0 f0 50 01 00 00 00 00 00 00 00 00 00 00 36 09 01 00 00 dd 00 00 24 00 00 00 00 dd 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 cc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 4c 4f 1b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00");
for (i = 0; i < 16; i++) { // bulldoze over header
try_fgetc(file);
}
while ((c = fgetc(file)) != EOF) {
switch (c) {
case 0xff:
printf("%02x ", 0x63); // wait 882 samples (50th of a second), a shortcut for 0x61 0x72 0x03
break;
case 0xfe:
param = try_fgetc(file);
for (i = 0; i < param; i++) {
printf("%02x ", 0x63); // wait 882 samples (50th of a second), a shortcut for 0x61 0x72 0x03
}
break;
case 0xfd:
printf("%02x ", 0x66); // end of sound data
break;
default:
param = try_fgetc(file);
printf("%02x %02x %02x ", 0xa0, c, param);
}
}
printf("\n");
}
