Scallops and deckles, and where they come from
Both edges are older than photography and both are borrowed from paper. Knowing what they are in the physical world is most of what you need to set them well on a screen.
Two edges with different ancestors
A scallop is a manufactured edge. It is what you get from a die, a rotary cutter or a pair of shaped shears: a repeat of identical arcs, evenly spaced, cut into or out of a straight line. Victorian carte-de-visite mounts had them, doilies have them, so do the paper cases under a cake and the edges of a great many greeting cards. The eye reads a scallop as something that was made in a factory and made carefully, which is exactly why it flatters a photograph of a child or a wedding.
A deckle is the opposite. It is the untrimmed edge of a sheet of hand-made paper, formed where the pulp thins out against the wooden frame — the deckle — that holds the mould. No two are alike, none is straight, and the irregularity is the whole point. Where a scallop says careful, a deckle says made by hand and left that way. Torn paper is the same family with less politeness: an edge pulled apart rather than formed.
That difference in ancestry is worth holding on to, because it decides which control matters. A scallop is judged on its regularity. A deckle is judged on whether its irregularity looks plausible.