19

Hey, I'm trying to create a pipeline that takes files from a flat directory source/ that contains files with names formatted as yyyy-mm-dd-some-file-name.md and processes them with the output being created in destination/yyyy/mm/dd/some-file-name.md. I managed to string together an ugly script that converts the source path to destination path, but the problem is that make says that there is no rule to make the target. Here is my code so far:

SOURCE := $(wildcard source/*.md)
DESTINATION := $(foreach f,SOURCE,destination/$(shell echo $(notdir $(f)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3\/\4.md/'))

$(DESTINATION): $(SOURCE)
	@mkdir -p $(dir $(shell echo $(notdir $@) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3/'))
	/bin/bash ./myscript.sh $< > $@

I believe that make sees that there are no target directories yet and just aborts the rule, but I'm not knowledgeable enough and just starting out with that tool. I would appreciate some guidance if any of you folks know a bit of Make.

you are viewing a single comment's thread
view the rest of the comments
[-] blamster19@programming.dev 2 points 1 week ago* (last edited 1 week ago)

Oh right, that was not the whole Makefile, only the part that caused problems. As @fruitcantfly@programming.dev pointed out, it breaks down with more SOURCE files and populates the newly created directories with only the first SOURCE file. After some more poking and reading I found a method that works by using define:

define rule
$(DESTINATION)/$(shell echo $(notdir $(1)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3\/\4.md/'): $(1) | $(DESTINATION)/$(shell echo $(notdir $(1)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3/')
	/bin/bash ./myscript.sh $$< > $$@
endef

$(foreach f,$(SOURCE),$(eval $(call rule,$(f))))

I feel like this is an abuse of Make, but it works. When is it a good idea to delegate the building process to other tools?

[-] eleijeep@piefed.social 2 points 1 week ago

Yes, I didn't notice that, I should have tested it more thoroughly!

Make works best when there's a simple relation between target filename and the dependency filenames, and when the output filenames are known ahead of time (or are easily derived from a list of inputs).

You can use Make to achieve what you want though, even when the relations are complex. The idiomatic pattern is to dynamically create your dependency lists and include them at runtime. Make supports this as a first-class feature and will handle re-evaluating any rules that are changed every time an included Makefile is (re-)generated.

For example:

SOURCE := $(wildcard source/*.md)
SDEPS := $(SOURCE:.md=.d)

include all.d
include ${SDEPS}

all.d: ${SDEPS}
	sed -i "s/^/all: /" all.d

${SDEPS}: %.d: %.md 
	DEST="destination/$$(basename $< | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3\/\4.md/')"; \
	printf "%s: $<\n" $${DEST} >$@; \
	printf "%s " $${DEST} >>all.d

%.md:
	mkdir -p $(dir $@)
	/bin/bash ./myscript.sh $< >$@

(I've tested this one and it seems to work)

But if you want to avoid any complex use of Make then you can push a lot of the logic into your script and then just create a timestamp file for each input as a dummy output to represent the fact that the script was run on each input. eg.

SOURCE := $(wildcard source/*.md)
SOURCE_TS := $(SOURCE:.md=.ts)

all: ${SOURCE_TS}

%.ts: %.md
	/bin/bash ./myscript.sh $<
	touch $@

This is a much simpler Makefile, but now you need to create the output filename, make the output directory and redirect the script output inside the script itself. The other drawback is that it's harder to use the output file as an input to another rule. You can use the timestamp file as an input but you won't have the actual filename available to send to a command without some more scripting.

this post was submitted on 16 Aug 2026
19 points (100.0% liked)

Programming

28283 readers
121 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 3 years ago
MODERATORS