We study operator algebra of extended conformal defects in more than two spacetime dimensions. Such algebra structure encodes the combined effect of multiple impurities on physical observables at long distances as well as the interactions among the impurities. These features are formalized by a fusion product which we define for a pair of defects, after isolating divergences that capture the effective potential between the defects. These divergences generalize the usual Casimir energy and define a novel effective field theory for defects. We discuss general properties of the corresponding fusion algebra and give explicit examples. We also derive universal formulas for defect structure constants from the defect effective field theory.